Skip to content

Sample tito_session_mismatch instead of running it on every record fetch#63

Open
nightlessbaron wants to merge 1 commit into
prodfrom
feat/tito-mismatch-sample-rate
Open

Sample tito_session_mismatch instead of running it on every record fetch#63
nightlessbaron wants to merge 1 commit into
prodfrom
feat/tito-mismatch-sample-rate

Conversation

@nightlessbaron

Copy link
Copy Markdown

On agentic runs, get_session re-tokenizes the full trajectory to compute tito_session_mismatch on every record fetch, synchronously on the session-server event loop. At 64k-context SWE scale this blocks record retrieval — the client-side collect_records GET times out (120s), returns empty, and the sample gets marked ABORTED. On a recent run that was ~19k timeouts, ~58% of samples per step, which cascaded into whole groups being dropped by the infra-failure filter.

The mismatch is a diagnostic canary for chat-template / tokenizer drift, logged as tito_session_mismatch_rate. It is not a training-correctness guard — training tokens come straight from the inference response (prompt_token_ids / output_token_logprobs), and the per-turn alignment asserts on the training path already enforce that. So computing it on every session is pure overhead once the template is stable.

This gates it behind --tito-session-mismatch-sample-rate (default 0.0, off). Set 0.02 for a cheap drift signal, 1.0 to restore the previous always-on behaviour. Sampling is a deterministic crc32(session_id) so there is no RNG on the hot path.

…fetch

Re-tokenizing the full trajectory on the session-server event loop blocks record
retrieval at long context; gate it behind --tito-session-mismatch-sample-rate (default off).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@nightlessbaron nightlessbaron requested a review from a team July 1, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants